cuda: make graph-cache eviction reachable; qwen3_asr: survive concurrent-load memory pressure - #293

RTF data, audiocpp_cli --metrics as in the #276 review. Same slice as the PR description (12 GB H100L vGPU, CUDA backend). Method: one --request-sequence session per family per binary — 6 requests per audio length, first request per length excluded as graph build, p50 (min–max) of the 5 warm runs. Baseline is main @ 54ea3d1, patched is this branch.

qwen3_asr 0.6B Q8 (the family this PR touches):

nemotron_asr 0.6B Q8 (regression check for the shared ggml/framework changes):

Long-file RTF is unchanged on both families and nemotron sits inside its baseline spread everywhere. Short/medium qwen is measurably faster on the patched build (2 s: 0.0326 → 0.0258; ranges don't overlap) — plausibly the reset-before-rebuild change relieving allocator pressure, though I haven't isolated it; no length is slower. Consistent with the patch adding no code to the warm path (eviction runs at graph destruction, the pool trim only on a previously-fatal allocation failure).

@derekja Could you clean up the churn in the PR caused by the CRLF changes? Changes to GGML are usually high-risk because of their large blast radius and generally require a regression sweep, unless they’re purely additive and opt-in. The current PR shows full-file churn, which makes it difficult to tell what actually changed.

you bet, thanks. running regression tests overnight and will post when complete.

Thanks for the careful look, and apologies for the churn — that was my tooling silently normalizing line endings on the three external/ggml files (they're stored CRLF, so a text-mode edit rewrote every line). I've rebuilt both commits from main's exact bytes with only the real insertions and force-pushed; the PR now shows the actual change: +110/−22 across 7 files.

On blast radius, agreed — here's the breakdown and the regression sweep.

What the ggml side actually changes (+29 lines, no deletions): two proc-address table entries, ggml_backend_cuda_trim_pools() (new, reachable only via the new proc entry), a clear() override on the legacy pool that calls its existing clear_pool(), and a virtual void clear() {} with a default no-op body on the pool base. Nothing existing is modified. The one behavioral change to pre-existing code is the point of the fix: release_backend_graph_resources — already called by every family's graph destructors — stops being a silent no-op on CUDA, so the sweep below deliberately covers multiple families, not just qwen3_asr.

Regression sweep (12 GB H100L vGPU, CUDA backend, this branch):

• ctest (-DENGINE_BUILD_TESTS=ON): 56/56 passed.
• run_audiocpp_cli_path_tests.py + compare_audiocpp_cli_path_results.py, baseline main @ 54ea3d1 CLI vs patched CLI, every family I have local models for: qwen3_asr (3 cases), nemotron_asr (2), citrinet_asr (1), hviske_asr (1), voxcpm2 (2 — a TTS family, included since the eviction change fires in all families' destructors): 9/9 compared cases, 0 differences — artifacts and text byte-match between the two builds. Caveats for reproducibility: the resources/*.wav fixtures aren't in the repo, so I substituted local audio (identical for both runs, which is what the A/B needs); the qwen forced-aligner case exercises the aligner model I don't have installed and the voxcpm2 voice-clone case failed identically on both builds against my substitute reference clip — both failures are byte-identical base-vs-patched and counted in the 0-difference comparison.
• RTF: the --metrics A/B in the comment above (equal-or-faster, no length regressed).
• The concurrent-load soak from the PR description, re-run on the clean branch: previously-fatal workload self-heals; serial byte-stability unchanged.

One pre-existing note from running the unit suite: codec_dequant_parity (tests/moss_tts_local) failed to compile on the first full-tests build against current main headers — unrelated to this PR (neither file is in this diff); mentioning it in case it's already known.

Happy to run any additional sweep you'd like — the concurrent-load harness that produced the leak repro is scriptable and I can share it.

@derekja What worries me most is the ggml_backend_cuda_clear_graph change

if (strcmp(name, "ggml_backend_cuda_clear_graph") == 0) {
    return (void *)ggml_backend_cuda_clear_graph;
}

I agree this is a correctness fix: before this PR, release_backend_graph_resources(...) was effectively a no-op for CUDA. The concern is that this correctness fix can expose lifecycle/performance issues in models that destroy and rebuild graphs between requests. A common use case (mostly for DiT models) is sending the same request multiple times with only a different seed to find the best results. The graph shape does not change, so before this PR CUDA graph cache state could be reused accidentally -- a happy coincidence. E.g. I tested MiniMax-H3 text-to-audio with default settings in one server session, using repeated requests. The PR introduced about a 9% warmed-request regression in that case.

main warmed avg (across 5 requests): 4460.38 ms
PR warmed avg: 4858.80 ms
delta: +398.42 ms
regression: +8.93%

Depending on the GPU, model, and request shape, the CUDA graph rebuild cost may be small enough to look like noise, but it can also be noticeable.

So what could delay the merge now is that we need to audit which models are affected by this lifecycle change, then refactor and validate the affected models before merging the PR.

I think a safer approach is to make the new CUDA graph cleanup opt-in for now, with only Qwen3-ASR enabling it initially. Any thoughts?

Sure, that makes sense. I'm happy to take on some of the testing to make it more generalizable if you like, although I'd want some guidance on what could be most convincing in that regard. Alternatively, I could see the opt-in as an optional flag on the existing call, or by putting the new behaviour in a new call, evict_backend_graph_cache(backend, graph) or something like that.

Do you have a preference? I'll push something into this PR to optionalize it and re-run the tests, if you like.

@derekja Thanks! I’d prefer a simple opt-in flag on the existing call, something like:

void release_backend_graph_resources(
    ggml_backend_t backend,
    ggml_cgraph * graph,
    bool evict_cuda_graph_cache = false);

void release_backend_graph_resources(
    BackendType backend_type,
    ggml_backend_t backend,
    ggml_cgraph * graph,
    bool evict_cuda_graph_cache = false);

Then just:

void release_backend_graph_resources(
    ggml_backend_t backend,
    ggml_cgraph * graph,
    bool evict_cuda_graph_cache) {
    if (!evict_cuda_graph_cache) return; // existing callsite/behavior unchanged
     ....
}

Default should stay false, so existing models keep the current CUDA graph-cache behavior. Then Qwen3-ASR can opt into the real cleanup path in this PR

release_backend_graph_resources(backend, graph, true);

and we can audit/refactor other models separately later.

Done — implemented exactly as you sketched: evict_cuda_graph_cache = false on both overloads with the early return, so every existing call site keeps current behavior, and qwen3_asr's four graph destructors pass true. Pushed as one additional commit. Re-validated on this branch: ctest 56/56; the 5-family path-test comparison against main @54ea3d1 still reports 0 differences; and the qwen concurrent-load repro still self-heals with the opt-in in place. Happy to help with the wider audit whenever you get to it — the concurrent-load harness is yours if useful.

Thank you @derekja! PR merged 🎉